Low-dimensional Latent Representations Research Articles

GLDM: hit molecule generation with constrained graph latent diffusion model.

Discovering hit molecules with desired biological activity in a directed manner is a promising but profound task in computer-aided drug discovery. Inspired by recent generative AI approaches, particularly Diffusion Models (DM), we propose Graph Latent Diffusion Model (GLDM)-a latent DM that preserves both the effectiveness of autoencoders of compressing complex chemical data and the DM's capabilities of generating novel molecules. Specifically, we first develop an autoencoder to encode the molecular data into low-dimensional latent representations and then train the DM on the latent space to generate molecules inducing targeted biological activity defined by gene expression profiles. Manipulating DM in the latent space rather than the input space avoids complicated operations to map molecule decomposition and reconstruction to diffusion processes, and thus improves training efficiency. Experiments show that GLDM not only achieves outstanding performances on molecular generation benchmarks, but also generates samples with optimal chemical properties and potentials to induce desired biological activity.

Unsupervised spatially embedded deep representation of spatial transcriptomics

Optimal integration of transcriptomics data and associated spatial information is essential towards fully exploiting spatial transcriptomics to dissect tissue heterogeneity and map out inter-cellular communications. We present SEDR, which uses a deep autoencoder coupled with a masked self-supervised learning mechanism to construct a low-dimensional latent representation of gene expression, which is then simultaneously embedded with the corresponding spatial information through a variational graph autoencoder. SEDR achieved higher clustering performance on manually annotated 10 × Visium datasets and better scalability on high-resolution spatial transcriptomics datasets than existing methods. Additionally, we show SEDR’s ability to impute and denoise gene expression (URL: https://github.com/JinmiaoChenLab/SEDR/).

Comparing storm resolving models and climates via unsupervised machine learning

Global storm-resolving models (GSRMs) have gained widespread interest because of the unprecedented detail with which they resolve the global climate. However, it remains difficult to quantify objective differences in how GSRMs resolve complex atmospheric formations. This lack of comprehensive tools for comparing model similarities is a problem in many disparate fields that involve simulation tools for complex data. To address this challenge we develop methods to estimate distributional distances based on both nonlinear dimensionality reduction and vector quantization. Our approach automatically learns physically meaningful notions of similarity from low-dimensional latent data representations that the different models produce. This enables an intercomparison of nine GSRMs based on their high-dimensional simulation data (2D vertical velocity snapshots) and reveals that only six are similar in their representation of atmospheric dynamics. Furthermore, we uncover signatures of the convective response to global warming in a fully unsupervised way. Our study provides a path toward evaluating future high-resolution simulation data more objectively.

Link prediction by adversarial Nonnegative Matrix Factorization

Networks are now more crucial than ever for modeling complex systems with interconnected components. Many methods have been developed to infer unobserved links or predict latent links based on observed network topology. However, the effectiveness of current methods is often limited due to the nature of sparseness in large-scale networks. Nonnegative Matrix Factorization (NMF) is one of the most popular low-rank approximation methods which has been successfully applied to large-scale data. However, maintaining generalization from limited observations is still an open challenge. To overcome these challenges, we propose a novel link prediction method based on adversarial NMF, which reconstructs a sparse network by an efficient adversarial training algorithm. Unlike the conventional NMF methods, our model considers potential test adversaries beyond the pre-defined bounds and provides a robust reconstruction with good generalization power. Besides, to preserve the local structure of a network, we use the common neighbor algorithm to extract the node similarity and apply it to low-dimensional latent representation. Simultaneously, we use Frobenius-norm regularization to prevent the factorization from overfitting. We provide an effective Majorization-Minimization method to learn the model parameters. Our method outperforms the state-of-the-art methods, as shown by extensive experiment results on twelve real-world datasets.

DL-MSTO+: A deep learning-based multi-scale topology optimization framework via positive definiteness ensured material representation network

A multi-scale approach to topology optimization has recently emerged due to its lightweight, robust, and multi-functional characteristics. Considering material diversity, an increasing number of materials leads to a computational burden due to numerous design variables and homogenization equations. This study proposes DL-MSTO+, a deep learning-based multi-scale topology optimization framework. The framework consists of two distinct deep neural networks for multi-scale topology optimization problems to reduce the dimensionality of design variables and predict homogenized material properties. First, a generator network learns the low-dimensional representation of a material microstructure in an unsupervised manner. Thus, the trained generator produces a micro-structural image from a low-dimensional vector. Second, a predictor network is trained in a supervised manner to predict the homogenized elasticity matrix for a given micro-structural image. Additionally, the predictor is specifically designed to ensure the positive definiteness of the predicted elasticity matrix. Lastly, the networks are integrated into the algorithm to improve the efficiency of multi-scale topology optimization. The numerical experiments demonstrate higher efficiency for the proposed algorithm than the conventional multi-scale approach. Moreover, the proposed method provides connectable multi-scale designs, and the low-dimensional latent representation enables semantic interpolation between solutions.

A variational encoder–decoder approach to precise spectroscopic age estimation for large Galactic surveys

ABSTRACT Constraints on the formation and evolution of the Milky Way Galaxy require multidimensional measurements of kinematics, abundances, and ages for a large population of stars. Ages for luminous giants, which can be seen to large distances, are an essential component of studies of the Milky Way, but they are traditionally very difficult to estimate precisely for a large data set and often require careful analysis on a star-by-star basis in asteroseismology. Because spectra are easier to obtain for large samples, being able to determine precise ages from spectra allows for large age samples to be constructed, but spectroscopic ages are often imprecise and contaminated by abundance correlations. Here we present an application of a variational encoder–decoder on cross-domain astronomical data to solve these issues. The model is trained on pairs of observations from APOGEE and Kepler of the same star in order to reduce the dimensionality of the APOGEE spectra in a latent space while removing abundance information. The low dimensional latent representation of these spectra can then be trained to predict age with just ∼1000 precise seismic ages. We demonstrate that this model produces more precise spectroscopic ages ($\sim 22~{{\ \rm per\ cent}}$ overall, $\sim 11~{{\ \rm per\ cent}}$ for red-clump stars) than previous data-driven spectroscopic ages while being less contaminated by abundance information (in particular, our ages do not depend on [α/M]). We create a public age catalogue for the APOGEE DR17 data set and use it to map the age distribution and the age-[Fe/H]-[α/M] distribution across the radial range of the Galactic disc.

Input Distribution Coverage: Measuring Feature Interaction Adequacy in Neural Network Testing

Testing deep neural networks (DNNs) has garnered great interest in the recent years due to their use in many applications. Black-box test adequacy measures are useful for guiding the testing process in covering the input domain. However, the absence of input specifications makes it challenging to apply black-box test adequacy measures in DNN testing. The Input Distribution Coverage (IDC) framework addresses this challenge by using a variational autoencoder to learn a low dimensional latent representation of the input distribution, and then using that latent space as a coverage domain for testing. IDC applies combinatorial interaction testing on a partitioning of the latent space to measure test adequacy. Empirical evaluation demonstrates that IDC is cost-effective, capable of detecting feature diversity in test inputs, and more sensitive than prior work to test inputs generated using different DNN test generation methods. The findings demonstrate that IDC overcomes several limitations of white-box DNN coverage approaches by discounting coverage from unrealistic inputs and enabling the calculation of test adequacy metrics that capture the feature diversity present in the input space of DNNs.

Long–short-term memory encoder–decoder with regularized hidden dynamics for fault detection in industrial processes

The ability of recurrent neural networks (RNN) to model nonlinear dynamics of high dimensional process data has enabled data-driven RNN-based fault detection algorithms. Previous studies have focused on detecting faults by identifying the discrepancies in data distribution between the faulty and normal data, as reflected in prediction errors generated by RNN models. However, in industrial processes, variations in data distribution can also result from changes in normal control setpoints and compensatory control adjustments in response to disturbances, making it hard to differentiate between normal and faulty conditions. This paper proposes a fault detection method utilizing a long short-term memory (LSTM) encoder–decoder structure with regularized hidden dynamics and reversible instance normalization (RevIN) to compactly represent high-dimensional measurements for effective monitoring. During training, the hidden states of the model are regularized to form a low-dimensional latent space representation of the original multivariate time series data. As a result, the prediction errors of the latent states can be used to monitor the abnormal dynamic variations, while the reconstruction errors of the measured variables are used to monitor the abnormal static variations. Furthermore, the proposed indices can reflect operating conditions, even when the distribution of test data changes, which helps distinguish faults from normal adjustments and disturbances that controllers can settle. Data from numerical simulation and the Tennessee Eastman process are used to illustrate the effectiveness of the proposed fault detection method.

BmVAE: a variational autoencoder method for clustering single-cell mutation data.

Genetic intra-tumor heterogeneity (ITH) characterizes the differences in genomic variations between tumor clones, and accurately unmasking ITH is important for personalized cancer therapy. Single-cell DNA sequencing now emerges as a powerful means for deciphering underlying ITH based on point mutations of single cells. However, detecting tumor clones from single-cell mutation data remains challenging due to the error-prone and discrete nature of the data. We introduce bmVAE, a bioinformatics tool for learning low-dimensional latent representation of single cell based on a variational autoencoder and then clustering cells into subpopulations in the latent space. bmVAE takes single-cell binary mutation data as inputs, and outputs inferred cell subpopulations as well as their genotypes. To achieve this, the bmVAE framework is designed to consist of three modules including dimensionality reduction, cell clustering and genotype estimation. We assess the method on various synthetic datasets where different factors including false negative rate, data size and data heterogeneity are considered in simulation, and further demonstrate its effectiveness on two real datasets. The results suggest bmVAE is highly effective in reasoning ITH, and performs competitive to existing methods. bmVAE is freely available at https://github.com/zhyu-lab/bmvae. Supplementary data are available at Bioinformatics online.

ScSSA: A clustering method for single cell RNA-seq data based on semi-supervised autoencoder

Single cell sequencing is a technology for high-throughput sequencing analysis of genome, transcriptome and epigenome at the single cell level. It can improve the shortcomings of traditional methods, reveal the gene structure and gene expression state of a single cell, and reflect the heterogeneity between cells. Among them, the clustering analysis of single-cell RNA data is a very important step, but the clustering of single-cell RNA data is faced with two difficulties, dropout events and dimension curse. At present, many methods are only driven by data, and do not make full use of the existing biological information. In this work, we propose scSSA, a clustering model based on semi-supervised autoencoder, fast independent component analysis (FastICA) and Gaussian mixture clustering. Firstly, the semi-supervised autoencoder imputes and denoises the scRNA-seq data, and then get the low-dimensional latent representation. Secondly, the low-dimensional representation is reduced the dimension and clustered by FastICA and Gaussian mixture model respectively. Finally, scSSA is compared with Seurat, CIDR and other methods on 10 public scRNA-seq datasets. The results show that scSSA has superior performance in cell clustering on 10 public datasets. In conclusion, scSSA can accurately identify the cell types and is generally applicable to all kinds of single cell datasets. scSSA has great application potential in the field of scRNA-seq data analysis. Details in the code have been uploaded to the website https://github.com/houtongshuai123/scSSA/.

Multi-view spectral clustering with latent representation learning for applications on multi-omics cancer subtyping.

Driven by multi-omics data, some multi-view clustering algorithms have been successfully applied to cancer subtypes prediction, aiming to identify subtypes with biometric differences in the same cancer, thereby improving the clinical prognosis of patients and designing personalized treatment plan. Due to the fact that the number of patients in omics data is much smaller than the number of genes, multi-view spectral clustering based on similarity learning has been widely developed. However, these algorithms still suffer some problems, such as over-reliance on the quality of pre-defined similarity matrices for clustering results, inability to reasonably handle noise and redundant information in high-dimensional omics data, ignoring complementary information between omics data, etc. This paper proposes multi-view spectral clustering with latent representation learning (MSCLRL) method to alleviate the above problems. First, MSCLRL generates a corresponding low-dimensional latent representation for each omics data, which can effectively retain the unique information of each omics and improve the robustness and accuracy of the similarity matrix. Second, the obtained latent representations are assigned appropriate weights by MSCLRL, and global similarity learning is performed to generate an integrated similarity matrix. Third, the integrated similarity matrix is used to feed back and update the low-dimensional representation of each omics. Finally, the final integrated similarity matrix is used for clustering. In 10 benchmark multi-omics datasets and 2 separate cancer case studies, the experiments confirmed that the proposed method obtained statistically and biologically meaningful cancer subtypes.

Non-Newtonian fluid simulation and reconstruction from monocular videos

In fluid simulation, the parameters of the constitutive model are difficult to predict accurately, which leads to the inconsistency between the simulation results and the visual effects of monocular videos. To solve this problem, a monocular video-guided non-Newtonian fluid reconstruction method is proposed. The model takes non-Newtonian fluid simulation videos as input for training and learns the best low-dimensional latent space representation of a single frame of fluid simulation images. Inter-frame prediction is then performed in this latent domain, employing a convolutional long–short-term memory network to predict latent vector representations of future frames. Finally, the reconstruction parameters of the constitutive model are predicted based on the frame-by-frame latent representation encoding and inter-frame temporal features. In the model verification stage, the real video of the non-Newtonian fluid is used as the input to predict the parameters of the fluid constitutive model, and realize the simulation and reconstruction of the non-Newtonian fluid based on the Cross model. Experimental results showed that the video-guided simulation reconstruction method could obtain fluid flow phenomena that were more consistent with those in the real video than the reconstruction method based on rheometer measurements. Furthermore, it provided higher pixel accuracy and precision at different times and visual effects that were more consistent with the actual fluid flow.

Trade-off between conservation of biological variation and batch effect removal in deep generative modeling for single-cell transcriptomics

BackgroundSingle-cell RNA sequencing (scRNA-seq) technology has contributed significantly to diverse research areas in biology, from cancer to development. Since scRNA-seq data is high-dimensional, a common strategy is to learn low-dimensional latent representations better to understand overall structure in the data. In this work, we build upon scVI, a powerful deep generative model which can learn biologically meaningful latent representations, but which has limited explicit control of batch effects. Rather than prioritizing batch effect removal over conservation of biological variation, or vice versa, our goal is to provide a bird’s eye view of the trade-offs between these two conflicting objectives. Specifically, using the well established concept of Pareto front from economics and engineering, we seek to learn the entire trade-off curve between conservation of biological variation and removal of batch effects.ResultsA multi-objective optimisation technique known as Pareto multi-task learning (Pareto MTL) is used to obtain the Pareto front between conservation of biological variation and batch effect removal. Our results indicate Pareto MTL can obtain a better Pareto front than the naive scalarization approach typically encountered in the literature. In addition, we propose to measure batch effect by applying a neural-network based estimator called Mutual Information Neural Estimation (MINE) and show benefits over the more standard maximum mean discrepancy measure.ConclusionThe Pareto front between conservation of biological variation and batch effect removal is a valuable tool for researchers in computational biology. Our results demonstrate the efficacy of applying Pareto MTL to estimate the Pareto front in conjunction with applying MINE to measure the batch effect.

Self-supervised anomaly detection for new physics

We investigate a method of model-agnostic anomaly detection through studying jets, collimated sprays of particles produced in high-energy collisions. We train a transformer neural network to encode simulated QCD ``event space'' dijets into a low-dimensional ``latent space'' representation. We optimize the network using the self-supervised contrastive loss, which encourages the preservation of known physical symmetries of the dijets. We then train a binary classifier to discriminate a beyond the standard model resonant dijet signal from a QCD dijet background both in the event space and the latent space representations. We find the classifier performances on the event and latent spaces to be comparable. We finally perform an anomaly detection search using a weakly supervised bump hunt on the latent space dijets, finding again a comparable performance to a search run on the physical space dijets. This opens the door to using low-dimensional latent representations as a computationally efficient space for resonant anomaly detection in generic particle collision events.

A Method with Adaptive Graphs to Constrain Multi-View Subspace Clustering of Geospatial Big Data from Multiple Sources

Clustering of multi-source geospatial big data provides opportunities to comprehensively describe urban structures. Most existing studies focus only on the clustering of a single type of geospatial big data, which leads to biased results. Although multi-view subspace clustering methods are advantageous for fusing multi-source geospatial big data, exploiting a robust shared subspace in high-dimensional, non-uniform, and noisy geospatial big data remains a challenge. Therefore, we developed a method with adaptive graphs to constrain multi-view subspace clustering of multi-source geospatial big data (agc2msc). First, for each type of data, high-dimensional and noisy original features were projected into a low-dimensional latent representation using autoencoder networks. Then, adaptive graph constraints were used to fuse the latent representations of multi-source data into a shared subspace representation, which preserved the neighboring relationships of data points. Finally, the shared subspace representation was used to obtain the clustering results by employing a spectral clustering algorithm. Experiments on four benchmark datasets showed that agc2msc outperformed nine state-of-the-art methods. agc2msc was applied to infer urban land use types in Beijing using the taxi GPS trajectory, bus smart card transaction, and points of interest datasets. The clustering results may provide useful calibration and reference for urban planning.

Hospital-Agnostic Image Representation Learning in Digital Pathology.

Whole Slide Images (WSIs) in digital pathology are used to diagnose cancer subtypes. The difference in procedures to acquire WSIs at various trial sites gives rise to variability in the histopathology images, thus making consistent diagnosis challenging. These differences may stem from variability in image acquisition through multi-vendor scanners, variable acquisition parameters, and differences in staining procedure; as well, patient demographics may bias the glass slide batches before image acquisition. These variabilities are assumed to cause a domain shift in the images of different hospitals. It is crucial to overcome this domain shift because an ideal machine-learning model must be able to work on the diverse sources of images, independent of the acquisition center. A domain generalization technique is leveraged in this study to improve the generalization capability of a Deep Neural Network (DNN), to an unseen histopathology image set (i.e., from an unseen hospital/trial site) in the presence of domain shift. According to experimental results, the conventional supervisedlearning regime generalizes poorly to data collected from different hospitals. However, the proposed hospital-agnostic learning can improve the generalization considering the lowdimensional latent space representation visualization, and classification accuracy results.

Learning representations of chromatin contacts using a recurrent neural network identifies genomic drivers of conformation

Despite the availability of chromatin conformation capture experiments, discerning the relationship between the 1D genome and 3D conformation remains a challenge, which limits our understanding of their affect on gene expression and disease. We propose Hi-C-LSTM, a method that produces low-dimensional latent representations that summarize intra-chromosomal Hi-C contacts via a recurrent long short-term memory neural network model. We find that these representations contain all the information needed to recreate the observed Hi-C matrix with high accuracy, outperforming existing methods. These representations enable the identification of a variety of conformation-defining genomic elements, including nuclear compartments and conformation-related transcription factors. They furthermore enable in-silico perturbation experiments that measure the influence of cis-regulatory elements on conformation.

ZINB-Based Graph Embedding Autoencoder for Single-Cell RNA-Seq Interpretations

Single-cell RNA sequencing (scRNA-seq) provides high-throughput information about the genome-wide gene expression levels at the single-cell resolution, bringing a precise understanding on the transcriptome of individual cells. Unfortunately, the rapidly growing scRNA-seq data and the prevalence of dropout events pose substantial challenges for cell type annotation. Here, we propose a single-cell model-based deep graph embedding clustering (scTAG) method, which simultaneously learns cell–cell topology representations and identifies cell clusters based on deep graph convolutional network. scTAG integrates the zero-inflated negative binomial (ZINB) model into a topology adaptive graph convolutional autoencoder to learn the low-dimensional latent representation and adopts Kullback–Leibler (KL) divergence for the clustering tasks. By simultaneously optimizing the clustering loss, ZINB loss, and the cell graph reconstruction loss, scTAG jointly optimizes cluster label assignment and feature learning with the topological structures preserved in an end-to-end manner. Extensive experiments on 16 single-cell RNA-seq datasets from diverse yet representative single-cell sequencing platforms demonstrate the superiority of scTAG over various state-of-the-art clustering methods.

Variational Autoencoders for Dimensionality Reduction of Automotive Vibroacoustic Models

<div class="section abstract"><div class="htmlview paragraph">In order to predict reality as accurately as possible leads to the fact that numerical models in automotive vibroacoustic problems become increasingly high dimensional. This makes applications with a large number of model evaluations, e.g. optimization tasks or uncertainty quantification hard to solve, as they become computationally very expensive. Engineers are thus faced with the challenge of making decisions based on a limited number of model evaluations, which increases the need for data-efficient methods and reduced order models.</div><div class="htmlview paragraph">In this contribution, variational autoencoders (VAEs) are used to reduce the dimensionality of the vibroacoustic model of a vehicle body and to find a low-dimensional latent representation of the system. Autoencoders are neural networks consisting of an encoder and a decoder network and they are trained in order to learn the identity mapping between a reduced approximation and the initial input while enforcing a dimensionality reduction in the latent space. This allows decoding the hidden data generating structure behind the data and enables an interpretation based on the latent variables, which is extremely valuable in the engineering design process. The performance of the VAE approach is compared to a conventional principal component analysis. Finally, the trained VAE is used as a deep generative model and it is investigated to which extent the pre-trained decoder network can be used to generate new artificial realizations at low costs. These artificially generated samples can then be used to enhance the training data basis for other neural network approaches or data-driven surrogate models.</div></div>

Searching for Anomalies in the ZTF Catalog of Periodic Variable Stars

Periodic variables illuminate the physical processes of stars throughout their lifetime. Wide-field surveys continue to increase our discovery rates of periodic variable stars. Automated approaches are essential to identify interesting periodic variable stars for multiwavelength and spectroscopic follow-up. Here we present a novel unsupervised machine-learning approach to hunt for anomalous periodic variables using phase-folded light curves presented in the Zwicky Transient Facility Catalogue of Periodic Variable Stars by Chen et al. We use a convolutional variational autoencoder to learn a low-dimensional latent representation, and we search for anomalies within this latent dimension via an isolation forest. We identify anomalies with irregular variability. Most of the top anomalies are likely highly variable red giants or asymptotic giant branch stars concentrated in the Milky Way galactic disk; a fraction of the identified anomalies are more consistent with young stellar objects. Detailed spectroscopic follow-up observations are encouraged to reveal the nature of these anomalies.